A Survey on Online Judge Systems and Their Applications

摘要

Online judges are systems designed for the reliable evaluation of algorithmsource code submitted by users, which is next compiled and tested in ahomogeneous environment. Online judges are becoming popular in variousapplications. Thus, we would like to review the state of the art for thesesystems. We classify them according to their principal objectives into systemssupporting organization of competitive programming contests, enhancingeducation and recruitment processes, facilitating the solving of data miningchallenges, online compilers and development platforms integrated as componentsof other custom systems. Moreover, we introduce a formal definition of anonline judge system and summarize the common evaluation methodology supportedby such systems. Finally, we briefly discuss an Optil.io platform as an exampleof an online judge system, which has been proposed for the solving of complexoptimization problems. We also analyze the competition results conducted usingthis platform. The competition proved that online judge systems, strengthenedby crowdsourcing concepts, can be successfully applied to accurately andefficiently solve complex industrial- and science-driven challenges.